-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix deadloop if proxy_handle_upstream exit early than proxy_handle_downstream #480
base: main
Are you sure you want to change the base?
Conversation
pingora-proxy/src/proxy_h1.rs
Outdated
ctx, | ||
) | ||
.await?; | ||
downstream_state.maybe_finished(request_done); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We initialize downstream_state
with DownstreamStateMachine::new(session.as_mut().is_body_done());
and is_body_done()
already checks if the body is empty or done. This downstream_state.maybe_finished(request_done);
looks like a no-op to me?
Would you be able to provide example code that reproduces this problem or additional logs from when it occurred? Thanks. |
It's hard to reproduce, we collect about 17G log to figure out what happen at that points. From our limited not completed logs. It's happen in HTTP2 POST, retrying. content-length == 0 session.as_mut().is_body_done() == false
// false || true
buffer.is_some() || session.as_mut().is_body_empty()
// false || true
let end_of_body = end_of_body || data.is_none(); |
Ok, that’s helpful, I’ll spend some more time looking into this. |
I'm able to reproduce this race condition locally, this was a good hint. The Returning the |
yes, end of stream has different means than end of body. I know that's not the best solution but after patch it, it's never happened again in our prod. you can give some hint how I could change pr. |
We'll want to fix It's also a good idea in the This is a bit more involved, we can fix this internally if you don't have time to amend the PR. Thanks again for filing an issue for this. |
Yes, we can revert proxy_h2 changes and also we haven't use h2 upstream yet. |
fix #475